Skip to content

Release agents while awaiting human clarification#79

Merged
khaliqgant merged 4 commits into
mainfrom
codex/issue-77-release-wake
Jul 16, 2026
Merged

Release agents while awaiting human clarification#79
khaliqgant merged 4 commits into
mainfrom
codex/issue-77-release-wake

Conversation

@miyaontherelay

Copy link
Copy Markdown
Contributor

Summary

  • release the full agent team after a durable Slack clarification is reserved and fleet absence is confirmed
  • wake exactly once from the first human reply with resumable sessions, cold-start context fallback, durable leases, and per-agent progress
  • restore reply watchers after restart, cancel stale issue wakes, tag configured stakeholders, and retry seven-day escalations durably

Validation

  • npm run build
  • npm test -- --run (37 files, 695 tests)
  • focused config/templates/state/factory suite (287 tests)
  • npm pack --dry-run
  • git diff --check
  • independent Codex shadow review: sign-off, no blockers

Closes #77

@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@miyaontherelay, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 53 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a2641e8a-eb7b-48a9-b3db-f36ede99e2db

📥 Commits

Reviewing files that changed from the base of the PR and between 99611d8 and 76e5fe6.

📒 Files selected for processing (10)
  • src/config/schema.test.ts
  • src/config/schema.ts
  • src/dispatch/templates.test.ts
  • src/dispatch/templates.ts
  • src/orchestrator/factory.test.ts
  • src/orchestrator/factory.ts
  • src/ports/state.ts
  • src/state/file-state-store.test.ts
  • src/state/file-state-store.ts
  • src/state/in-memory-state-store.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/issue-77-release-wake

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a durable clarification parking and wake mechanism for agent teams when they require human input mid-task. It adds support for Slack stakeholder mentions, state persistence for waiting clarifications, and automated escalations for unanswered questions. The review feedback highlights two main areas for improvement: first, handling transient errors during background lease renewal more gracefully to prevent premature wake aborts; second, optimizing the agent release process by reducing redundant fleet roster queries.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +5154 to +5160
const heartbeat = setInterval(() => {
if (renewalInFlight || leaseLost) return
renewalInFlight = true
void renewLease()
.catch(() => { leaseLost = true })
.finally(() => { renewalInFlight = false })
}, Math.max(1_000, Math.floor(CLARIFICATION_WAKE_LEASE_MS / 3)))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

A transient network or database error during the background heartbeat will cause renewLease() to throw an exception. Currently, the .catch(() => { leaseLost = true }) block on the heartbeat interval catches any error and permanently sets leaseLost = true. This will prematurely abort the entire wake process even if the network recovers immediately and the lease hasn't actually expired or been claimed by another daemon.

We should only set leaseLost = true if the lease is definitely lost (i.e., when renewClarificationWake returns false and throws ClarificationWakeLeaseLostError). Other transient errors should be logged and retried on the next heartbeat tick.

    const heartbeat = setInterval(() => {
      if (renewalInFlight || leaseLost) return
      renewalInFlight = true
      void renewLease()
        .catch((error) => {
          if (error instanceof ClarificationWakeLeaseLostError) {
            leaseLost = true
          } else {
            this.#logger.warn?.('[factory] transient error renewing clarification wake lease', error)
          }
        })
        .finally(() => { renewalInFlight = false })
    }, Math.max(1_000, Math.floor(CLARIFICATION_WAKE_LEASE_MS / 3)))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 75db518. The heartbeat now treats only ClarificationWakeLeaseLostError as confirmed ownership loss; transient state-store errors are logged and retried on later heartbeats. Explicit renewals around resume, spawn, injection, and completion still fail closed. Added a timed blocked-resume regression that injects one transient background renewal failure and proves the team completes its wake without a lease-loss event. Full suite: 696/696.

Comment on lines +3187 to 3219
async #releaseAgentsForClarification(key: string, agents: Array<[string, TrackedAgent]>): Promise<void> {
let waiting = await this.#state.getWaitingClarification(this.#workspaceId, key)
if (!waiting) return
for (const [name, tracked] of agents) {
const onlineBefore = new Set((await this.#fleet.roster()).agents.map((agent) => agent.name))
if (waiting.releasedAgents?.includes(name) && !onlineBefore.has(name)) continue
try {
// Prefer broker release over process termination so the harness gets a
// graceful shutdown boundary and can flush its latest resumable state.
await this.#fleet.release(name, 'waiting-for-human')
} catch (error) {
this.#logger.warn?.('[factory] graceful clarification release failed; forcing local teardown', {
agentName: name,
error,
})
await this.#releaseAndTerminateAgents([[name, tracked]], 'waiting-for-human', 'clarification')
}
const onlineAfter = new Set((await this.#fleet.roster()).agents.map((agent) => agent.name))
if (onlineAfter.has(name)) {
throw new Error(`fleet still reports ${name} online after clarification release`)
}
waiting = await this.#state.markClarificationAgentReleased(this.#workspaceId, key, name) ?? waiting
}

// Check the whole snapshot once more before opening the wake gate. This
// catches server-side restart policies that re-register a name between its
// individual release confirmation and the final parked transition.
const online = new Set((await this.#fleet.roster()).agents.map((agent) => agent.name))
const stillOnline = agents.map(([name]) => name).filter((name) => online.has(name))
if (stillOnline.length > 0) {
throw new Error(`clarification agents still online: ${stillOnline.join(', ')}`)
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The #releaseAgentsForClarification method fetches the fleet roster multiple times inside the loop (both before and after releasing each agent) and once more at the end. Since fetching the roster typically involves an external API call or container orchestrator query, doing this repeatedly can be highly inefficient and slow down the release process.

We can optimize this by fetching the roster once at the start of the method, updating our local online set as we release agents, and only re-fetching the roster when an agent is actually released to verify its status.

  async #releaseAgentsForClarification(key: string, agents: Array<[string, TrackedAgent]>): Promise<void> {
    let waiting = await this.#state.getWaitingClarification(this.#workspaceId, key)
    if (!waiting) return
    const roster = await this.#fleet.roster()
    let online = new Set(roster.agents.map((agent) => agent.name))

    for (const [name, tracked] of agents) {
      if (waiting.releasedAgents?.includes(name) && !online.has(name)) continue
      try {
        // Prefer broker release over process termination so the harness gets a
        // graceful shutdown boundary and can flush its latest resumable state.
        await this.#fleet.release(name, 'waiting-for-human')
      } catch (error) {
        this.#logger.warn?.('[factory] graceful clarification release failed; forcing local teardown', {
          agentName: name,
          error,
        })
        await this.#releaseAndTerminateAgents([[name, tracked]], 'waiting-for-human', 'clarification')
      }
      const onlineAfter = new Set((await this.#fleet.roster()).agents.map((agent) => agent.name))
      if (onlineAfter.has(name)) {
        throw new Error(`fleet still reports ${name} online after clarification release`)
      }
      online = onlineAfter
      waiting = await this.#state.markClarificationAgentReleased(this.#workspaceId, key, name) ?? waiting
    }

    // Check the whole snapshot once more before opening the wake gate. This
    // catches server-side restart policies that re-register a name between its
    // individual release confirmation and the final parked transition.
    const stillOnline = agents.map(([name]) => name).filter((name) => online.has(name))
    if (stillOnline.length > 0) {
      throw new Error(`clarification agents still online: ${stillOnline.join(', ')}`)
    }
  }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optimized in 75db518. The method now takes one initial roster snapshot and refreshes it only after an actual release, eliminating the redundant pre-release query for each agent. I deliberately retained the final independent roster fetch: reusing the last per-agent snapshot would miss an earlier agent re-registering between its checkpoint and the parked transition, which is a core safety invariant for this issue.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found and verified against the latest diff

You’re at about 99% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/state/file-state-store.ts">

<violation number="1" location="src/state/file-state-store.ts:140">
P1: A claimed Slack answer can remain permanently parked after a daemon crash because this line durably records `reply` before the wake lease is claimed. Startup recovery would be safer if it drained persisted replies or atomically persisted the reply and wake intent instead of skipping records that already contain `waiting.reply`.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread src/orchestrator/factory.ts Outdated
if (!record || record.reply) {
return undefined
}
record.reply = { ...reply }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: A claimed Slack answer can remain permanently parked after a daemon crash because this line durably records reply before the wake lease is claimed. Startup recovery would be safer if it drained persisted replies or atomically persisted the reply and wake intent instead of skipping records that already contain waiting.reply.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/state/file-state-store.ts, line 140:

<comment>A claimed Slack answer can remain permanently parked after a daemon crash because this line durably records `reply` before the wake lease is claimed. Startup recovery would be safer if it drained persisted replies or atomically persisted the reply and wake intent instead of skipping records that already contain `waiting.reply`.</comment>

<file context>
@@ -71,9 +77,273 @@ export class FileStateStore extends InMemoryStateStore {
+        if (!record || record.reply) {
+          return undefined
+        }
+        record.reply = { ...reply }
+        await this.#persist(document)
+        return cloneClarification(record)
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reported permanent-park path was not present: startup and the run loop already drain durable records containing reply even without a new Slack event. I added the exact crash-window regression in 6844d1a: it seeds a fully parked FileStateStore record with a persisted reply but no wake lease, constructs a fresh factory with no new Slack event, and proves startup resumes/injects the team exactly once and clears the record.

Comment thread src/dispatch/templates.ts
@khaliqgant
khaliqgant merged commit 7815b7e into main Jul 16, 2026
3 checks passed
@khaliqgant
khaliqgant deleted the codex/issue-77-release-wake branch July 16, 2026 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Release-on-question, wake-on-reply: don't hold an agent's session while a human answers

2 participants